Papers with pretrain-finetune discrepancy
CSP:Code-Switching Pre-training for Neural Machine Translation (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to train neural machine translation models are data-hungry and low-resource . et al., 2018; Radford e.t., 2019; Yang ee.,2019) proposes a new pre-training method for NMT . |
| Approach: | They propose a new pre-training method which randomly replaces some words in the input sentence with their translation words in target language. |
| Outcome: | The proposed method improves on unsupervised and supervised NMT models by making full use of monolingual corpora. |
Rethinking Denoised Auto-Encoding in Language Pre-Training (2021.emnlp-main)
Copied to clipboard
| Challenge: | Pre-trained models such as BERT have achieved success in learning sequence representations, but they tend to learn representations that are covariant with the noise of pre-training. |
| Approach: | They propose to train self-trained models to learn noise invariant sequence representations . they encourage consistency between original sequence and corrupted version via unsupervised instance-wise training signals. |
| Outcome: | The proposed model improves on 11 natural language understanding and cross-modal tasks and achieves 0.6% gain on GLUE benchmarks and 0.8% increment on NLVR2 . |
Pseudo Zero Pronoun Resolution Improves Zero Anaphora Resolution (2021.emnlp-main)
Copied to clipboard
| Challenge: | Masked language models have contributed to drastic performance improvements with regard to zero anaphora resolution (ZAR). |
| Approach: | They propose a pretraining task that trains MLMs on anaphoric relations with explicit supervision and a finetuning method that remedies a notorious discrepancy. |
| Outcome: | The proposed method improves zero anaphora resolution in Japanese ZAR . it uses a pretrain task and finetuning task to correct the discrepancy . |
Transfer Learning for Sequence Generation: from Single-source to Multi-source (2021.acl-long)
Copied to clipboard
| Challenge: | Recent studies have shown that pretrained models are effective for low-resource downstream tasks. |
| Approach: | They propose a two-stage finetuning method to transfer pretrained models to MSG tasks by concatenating multiple sources into a single long sequence. |
| Outcome: | The proposed model outperforms baselines on the WMT17 APE task and multi-source translation task using the WTM14 test set. |
DiffusionRet: Diffusion-Enhanced Generative Retriever using Constrained Decoding (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Generative retrieval methods have suffered from the lack of the intermediate reasoning step . generative retrieval uses sequence-to-sequence diffusion models to map a query to relevant docids . |
| Approach: | They propose a novel method that uses query as an intermediate step before retrieval . they propose to use sequence-to-sequence diffusion models to map a query to relevant docids . |
| Outcome: | Experiments show that proposed method outperforms existing methods on MARCO and Natural Questions datasets. |